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1  BACKGROUND 

The  Human  Research  and  Engineering  Directorate  (HRED),  U.S.  Army  Research  Laboratory  (ARL)  5 -year 
Army  Technology  Objective  (ATO)  research  program  is  addressing  human  robot  interaction  (HRI)  and 
teaming  for  both  aerial  and  ground  robotic  assets  in  conjunction  with  the  U.S.  Army  Tank  and  Automotive 
Research  and  Development  Engineering  Center  (TARDEC).  The  ATO  has  recently  been  enlarged  to 
encompass  intelligent  collaboration  among  unmanned  aerial  systems  (UAS)  and  renamed  the  Collaborative 
Robotics  ATO.  The  purpose  of  the  program  is  to  understand  HRI  issues  in  order  to  develop  and  evaluate 
technologies  to  improve  HRI  battlefield  performance  for  Future  Combat  Systems  (FCS)  and  the  Future  Force 
Warrior  (FFW).  Soldier  robot  teams  will  be  an  important  component  of  future  battlespaces:  creating  a 
complex  but  potentially  more  survivable  and  effective  combat  force.  The  complexity  of  the  battlefield  of  the 
future  presents  its  own  problems.  The  variety  of  robotic  systems  and  the  almost  infinite  number  of  possible 
Army  missions  create  a  dilemma  for  researchers  who  wish  to  predict  HRI  performance  in  future 
environments.  Most  of  the  FCS  proposed  systems  are  still  in  the  conceptual  stage  and  the  nature  of  the 
environments  that  they  are  being  designed  for  can  only  be  approximated. 

1.1  Statement  of  the  Problem 

The  purpose  of  this  paper  is  to  demonstrate  how  current  virtual  technologies  are  being  used  to  answer  both 
very  applied  and  more  basic  questions  concerning  how  humans  and  robots  interact  and  form  teams  to  conduct 
missions  in  a  variety  of  future  combat  situations.  The  general  research  philosophy  is  to  conduct  the  research 
using  virtual  environments  that  are  comparatively  isomorphic  to  the  functional  requirements  of  future 
battlespaces.  We  argue  that,  in  some  cases,  trying  to  capture  the  exact  conditions  of  future  conflict  is  self- 
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defeating,  because  of  the  unpredictability  of  future  events.  Rather,  we  believe  it  is  more  efficient  to  sample  a 
variety  of  future  battlespaces  and  vary  the  functional  requirements  of  both  the  robotic  and  the  Soldier’s 
tasking.  Understanding  the  richness  of  possible  HRI  paradigms  should  allow  us  to  develop  general  models  of 
HRI  and  HR  teaming  performance  in  order  to  predict  operational  and  design  requirements  before  either  the 
robotic  systems  or  the  future  battle  environments  are  fully  articulated. 

1.2  Summary  of  the  Findings 

A  good  example  is  the  clever  use  of  the  ‘‘Tom  Clancy's  Rainbow  Six  3:  Raven  ShielcF^^''  gaming  software  by 
researchers  from  Micro  Analysis  &  Design  (MA&D)  and  HRED.  The  gaming  software  allowed  the 
researchers  to  investigate  Blackhawk  Down  type  rescue  missions  using  both  aerial  and  ground  robotic  systems 
in  an  urban  environment.  They  were  able  to  ask  generic  questions  about  collaborative  technologies  to 
investigate  different  HRI  teaming  relationships  using  fairly  realistic  depictions  of  urban  and  desert 
environments  while  capturing  the  essence  of  a  realistic  combat  mission. 

Perhaps  an  even  more  realistic  representation  of  an  actual  urban  combat  environment  was  created  by  Dr. 
Florian  Jentsch  and  his  colleagues  at  the  University  of  Central  Florida  (UCF).  They  fabricated  a  1/35  scale 
Iraqi  city  with  toy  buildings.  Mosques,  roads,  and  human  figures  in  realistic  Arab  dress  to  emulate  an  urban 
reconnaissance  mission  conducted  by  robotic  vehicles  controlled  remotely  by  military  operators.  The  purpose 
of  the  UCF  research  is  to  understand  teaming  relationships  between  multiple  operators  and  multiple  robotic 
assets  in  an  urban  environment.  The  scaled  city  has  proved  to  be  very  versatile  allowing  the  experimenters  to 
change  the  cityscape  easily  and  conduct  various  missions  without  requiring  software  changes.  Unbeknownst 
to  the  test  participants,  the  scaled  robots  themselves  are  controlled  by  confederate  researchers  simulating 
semiautonomous  Armed  Robotic  Vehicles  (ARV).  The  surrogate  operators  plan  the  waypoints  necessary  to 
conduct  their  mission  efficiently.  The  scaled  urban  environment  is  flexible  and  cost  effective  and  yet  has 
produced  valuable  data  concerning  the  effectiveness  of  intelligence  gathering  under  different  experimental 
conditions. 

Dr.  Raja  Parasuraman  of  George  Mason  University  and  Dr.  Keryl  Cosenzo  of  HRED  have  created  simulation 
environments  to  investigate  the  effects  automating  various  robotic  tasks  during  multi-tasking  mounted 
missions.  One  of  their  virtual  environments  represents  a  generic  multi-tasking  structure  for  a  potential  robotic 
non-commissioned  officer  (NCO)  and  the  research  focus  is  on  how  to  best  automate  various  functions  when 
the  operator  must  control  or  monitor  multiple  systems  including  UAS  and  unmanned  ground  vehicles  (UGVs) 
during  maneuver  missions  conducted  over  various  types  of  terrain.  Specifically,  they  are  investigating  the 
effects  of  adaptive  processes,  types  of  automated  control,  and  adaptable  planning.  Of  particular  interest  is  the 
fact  that  they  plan  to  use  a  number  of  inexpensive  simulation  environments  in  order  to  generalize  their  results. 

Last,  but  not  least.  Dr.  Jessie  Chen  from  HRED  and  her  colleagues  conducted  simulations  at  the  U.S.  Army 
Simulation  Training  Technology  Center  (STTC)  in  Orlando,  FL.  A  mock-up  of  a  generic  crew  station  with 
mission  software  was  constructed  by  UCF  engineers  to  investigate  workload  demands  of  future  robotic 
operators  conducting  realistic  FCS  reconnaissance  missions  with  a  limited  crew  size.  The  experimental 
question  was  whether  the  gunner  in  the  crew  station  could  conduct  remote  targeting  using  an  ARV  while 
concurrently  performing  his  or  her  primary  task  of  protecting  the  mounted  system  from  local  threats.  The 
simulations  required  sophisticated  software  to  emulate  the  different  experimental  conditions  requiring  the 
MCS  to  transverse  software  generated  terrain.  Aside  from  the  crew  interface,  an  ARE  sponsored  operator 
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tactical  control  unit  (TCU)  that  has  been  developed  for  actual  robotic  control  was  used  to  enhance  the 
simulations  real  world  validity. 

Except  for  the  last  case,  none  of  the  research  projects  required  expensive  simulators  and  yet  the  experimenters 
were  able  to  investigate  a  number  of  very  sophisticated  HRI  missions.  The  remaining  sections  of  the  paper 
will  examine  results  from  the  various  efforts  and  reinforce  our  statement  that  it  is  possible  to  lay  a  solid 
foundation  of  HRI  understanding  for  future  design  efforts  using  multiple  virtual  tools  before  either  specific 
robotic  systems  or  their  intended  environments  are  fully  realized. 


2  OFF-THE-SHELF  GAMING  SOFTWARE 

2.1  Background  and  Purpose 

This  research  investigated  how  teams  can  effectively  use  the  visually  rich  information  provided  by  unmanned 
vehicles  (UVs).  Robotic  assets  have  vast  potential  to  aid  Soldiers  by  providing  advance  warning  of  enemy 
activities,  precise  location  of  targets,  and  a  three-dimensional  understanding  of  terrain  including  potential 
avenues  of  approach. 

Two  experiments  were  conducted  to  investigate  real  world  issues  surrounding  the  use  of  UVs  and  UV- 
provided  information  such  as  team  configuration,  the  technologies  used,  and  the  information  that  was 
exchanged.  The  experiments  provided  insight  into  specific  questions  about  the  teaming  arrangement  between 
the  person  monitoring  the  UV  (referred  to  as  the  Information  Manager  or  IM)  and  the  Soldiers  in  the  field  who 
are  using  the  UV-provided  information  to  complete  a  mission  (referred  to  as  the  Rescue  Mission  player  or 
RM). 

•  Can  the  IM  be  positioned  rearward  at  a  command  center  or  should  he  or  she  be  co-located  with  soldiers 
in  the  field?  If  the  IM  is  in  a  command  post  he  or  she  could  better  concentrate  on  monitoring  the  UV  and 
determining  how  to  use  the  information  provided.  However,  this  may  come  at  a  cost  of  degraded 
situation  awareness  (SA)  and  understanding  of  which  information  is  most  useful  and  timely. 

•  How  useful  are  collaborative  technologies?  Does  it  help  to  have  access  to  an  electronic  map  that  shows 
the  dynamic  location  of  the  UV  and  the  Soldiers  in  the  field?  Does  the  ability  to  share  and  discuss  UV 
provided  images  or  the  electronic  map  improve  performance? 

•  If  being  co-located  is  superior,  what  is  it  about  being  co-located  that  results  in  better  performance?  Is  it 
visual  contact  between  the  team  members  coupled  with  the  ability  to  communicate  face  to  face  (FF),  the 
ability  to  share  and  discuss  visual  information,  or  the  ability  of  the  IM  to  see  what  the  Soldiers  in  the 
field  are  experiencing? 

2.2  Approach 

Tom  Clancy’s  Raven  Shield  computer  game  by  UbiSoft™  was  used  as  a  battlefield  simulator.  A  pair  of  U.S. 
Marines  worked  as  a  team  to  conduct  “Blackhawk  Down”  type  missions.  The  RM  navigated  to  a  downed 
helicopter  pilot  as  quickly  as  possible  without  being  detected  by  enemy  forces.  The  IM  used  a  UV  to  gather 
information  (such  as  the  crash  site,  safe  routes  to  the  site,  and  enemy  locations)  and  then  relayed  information 
to  the  RM.  The  IM  used  a  traditional  monitor  (19’  screen)  to  watch  the  video  from  the  UV,  while  the  RM  used 
a  large  5’  by  7’  screen  to  increase  immersion  in  the  game. 

The  first  experiment  (McDermott,  Luck,  Allender,  &  Fisher,  2005)  addressed  the  first  two  bulleted  issues 
above.  Researchers  manipulated  the  location  of  the  team  members,  the  method  of  communication,  and  the 
type  of  UV  asset  used:  air  (UAV)  or  ground  (UGV).  In  the  co-located  location,  the  IM  and  RM  were  in  the 
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same  room  and  the  IM  could  readily  see  the  RMs  view  of  the  simulated  world.  In  the  remote  location,  the  IM 
and  RM  were  in  different  rooms.  In  the  remote  with  map  location,  the  IM  could  also  toggle  his  or  her  screen 
to  view  an  electronic  map  with  the  UV  and  RM  locations.  Communication  was  either  verbal  only  (either  FF  or 
via  radio,  depending  on  the  location)  or  verbal  plus  visual.  When  visual  communication  was  allowed,  the  IM 
could  “send”  still  images  from  the  UV  or  the  electronic  map  to  the  RM.  This  was  done  via  a  small  monitor 
that  was  turned  to  face  the  RM  when  the  IM  wanted  to  transmit  an  image.  Both  UAVs  and  UGVs  were 
simulated  using  the  “ghost”  function  in  Raven  Shield  (see  Figure  1).  This  made  the  UV  invisible  and  allowed 
them  to  fly.  Because  the  ghost  function  also  enabled  the  UV  to  fly  through  walls  and  underground,  a 
confederate  controlled  the  UV  to  ensure  the  UV  behaved  functioned  in  a  realistic  manor.  The  dependent 
variables  included  objective  performance  metrics:  time,  task  completion,  detections  by  the  enemy,  and  the 
number  of  times  each  technology  resource  was  used. 


Figure  1:  View  of  Enemy  from  UGV  and  View  from  UAV  High  above  the  Terrain. 


Because  the  first  study  revealed  performance  advantages  to  being  co-located,  a  second  experiment  was 
conducted  to  understand  which  aspects  of  being  co-located  were  most  beneficial  (Luck,  McDermott,  Allender, 
&  Fisher,  2006;  Allender,  McDermott,  Luck,  &  Fisher,  2006).  In  this  experiment,  participants  would 
communicate  either  FF  or  through  a  visual  divider  placed  between  them.  Researchers  also  varied  whether  or 
not  the  IM  could  see  the  RMs  screen  and  whether  or  not  the  IM  could  transmit  visual  images  to  the  RM.  This 
experiment  environment  was  also  improved  by  giving  the  IM  two  monitors  so  the  IM  could  simultaneously 
see  the  real  time  UV  information  and  the  electronic  map  (see  Figure  2),  which  was  determined  to  be  an  issue 
in  Experiment  1 .  In  addition,  instead  of  using  the  ghost  function  for  the  UV,  we  shrank  a  simulated  player  in 
the  game  (Figure  2)  to  act  as  a  UGV,  so  that  the  UV  was  visible. 
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Figure  2:  The  RMs  View  with  the  UGV  in  the  Middie  of  the  Scene  and  the  Eiectronic  Map. 

2.3  What  Was  Learned 

The  first  experiment  revealed  several  interesting  findings.  Soldiers  were  detected  significantly  less  often  when 
they  were  aided  by  a  UV  than  when  they  completed  the  missions  alone.  However,  this  increased  safety  came 
at  a  price  of  higher  mission  times.  Teams  were  faster  and  detected  less  often  with  the  UGV  than  the  UAV. 
Despite  these  performance  benefits,  participants  preferred  the  flexibility  of  the  UAV  and  its  ability  to  provide 
the  big  picture.  Teams  were  faster  and  detected  less  often  with  Verbal  only  communication  than  the  Verbal 
plus  Visual  communication.  This  was  mainly  attributed  to  the  fact  that  the  transmitted  images  often  confused 
the  RM  more  than  it  helped  because  the  RM  did  not  have  a  good  reference  point  for  the  image.  Again,  the 
preferences  did  not  match  performance.  Participants  ranked  every  Verbal  plus  Visual  communication 
conditions  higher  than  any  Verbal  only  communication  condition.  Finally,  there  were  some  advantages  to 
being  co-located,  as  it  resulted  in  fastest  mission  completion  times  and  was  preferred  by  participants.  It  did 
not,  however,  decrease  the  number  of  detections  by  the  enemy. 

In  the  second  experiment,  performance  was  faster  when  participants  were  FF.  However,  closer  inspection 
revealed  that  this  only  held  true  when  image  transmission  was  not  allowed.  When  image  transmission  was 
allowed,  the  extrapolated  time  was  similar  in  FF  and  conditions  with  the  divider  (see  Figure  3).  The  number  of 
detections  by  the  enemy  followed  a  similar  pattern.  Therefore,  being  able  to  transmit  images  seemed  to 
compensate  for  the  fact  that  the  team  could  not  communicate  FF.  This  increase  in  performance  when 
transmitting  images  (compared  to  the  first  experiment  where  it  degraded  performance)  is  most  likely 
explained  by  the  fact  that  participants  had  more  training  in  the  appropriate  use  of  images  and  provided  a 
reference  point  for  the  RM  location  relative  to  the  image. 
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□  No  images,  Divider 

□  No  images,  FF 

□  Yes  images.  Divider 

□  Yes  images,  FF 


Figure  3:  Interaction  of  Ability  to  Transmit  Images  by  Communication. 


2.4  Advantages  and  Disadvantages 

Using  the  Raven  Shield  video  game  had  both  advantages  and  disadvantages.  The  “ghost”  mode  provided  the 
flexibility  to  fly  like  a  UAV  or  drive  like  a  UGV.  This  simulated  UV  was  relatively  easy  to  operate  using  a 
mouse  and  keyboard  with  minimal  training.  When  participants  controlled  the  UGV  in  Experiment  2,  they  did 
not  have  to  learn  how  to  make  complex  maneuvers  with  a  stick  and  throttle  or  flne  motor  inputs  with  a 
joystick.  This  allowed  researchers  to  focus  on  communication  issues,  not  control  issues.  In  addition,  the  maps 
in  Raven  Shield  were  tailorable.  A  variety  of  maps  were  available  on  the  Internet  and  a  high  level  of 
programming  was  not  needed  to  modify  the  location  of  terrorists  and  obstacles.  Multiple  forums  and 
chatrooms  exist  on  the  Internet  in  which  a  developer  can  learn  tricks  of  the  trade  and  ask  questions.  Lastly,  the 
Raven  Shield  software  was  inexpensive  compared  to  high  fldelity  simulators  and  robots.  The  software  itself 
was  $20-$30  and  the  only  other  equipment  required  was  upgraded  video  cards  at  approximately  $400  a 
computer. 

One  disadvantage  was  the  number  of  workarounds  required  to  get  the  game  to  work  as  intended.  For  example, 
the  “compass”  was  actually  a  tracking  device  which  tracked  a  dead  player  located  in  an  extreme  northern 
location.  Therefore,  the  compass  only  showed  where  north  was  relative  to  the  player’s  orientation  instead  of 
pointing  north  like  an  actual  compass.  It  also  took  quite  a  bit  of  manipulation  to  simultaneously  display  both 
the  electronic  map  and  the  UV  feed  to  the  IM.  This  was  eventually  accomplished  by  using  another  simulated 
player  whose  screen  was  used  to  show  the  map.  Another  workaround  (discovered  in  a  chatroom)  was  the 
“god”  feature  which  prevented  the  RM  from  being  killed  when  shot  at  by  the  enemy  and  prematurely  ending 
the  mission.  The  “god”  feature  also  allowed  the  UV  to  be  visible  to  the  RM.  All  these  workarounds  provided 
the  functionality  needed  but  they  made  development  more  difficult  and  required  more  time  to  set  up  each 
scenario  during  the  experiments.  Other  games  provide  more  flexibility  in  authoring  environments.  Perhaps  the 
biggest  disadvantage  was  the  fact  that  Raven  Shield  was  not  designed  to  collect  that  type  of  data  needed. 
Raven  Shield  does  provide  statistics  such  as  the  number  of  rounds  fired  and  the  number  of  enemy  killed  but 
these  were  not  part  of  our  scenario  objectives.  Therefore,  the  researchers  had  to  collect  data  manually  by 
taking  detailed  notes  and  watching  videotapes  of  the  experiment.  Despite  these  disadvantages,  the  researchers 
found  Raven  Shield  to  be  a  useful  tool  for  understanding  team  communications  in  human  robot  interactions. 
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3.1  Background  and  Purpose 

A  set  of  important  questions  in  the  area  of  HRI  has  been  related  to  the  way  in  which  multiple  operators 
interact  with  each  other  and  with  multiple  robots.  Although  the  goal  of  the  robotics  development  is  that  a 
single  soldier  would  have  the  ability  to  control  a  number  of  robotic  assets  at  the  same  time,  achieving 
operator-to-robot  ratios  at  or  below  1:1  has  proven  elusive.  While  controlling  multiple  robots  may  be  a 
relatively  easier  task  in  the  case  of  logistical  missions  (in  which  control  of  a  lead  vehicle  can  transfer  to 
trailing  vehicles  in  a  caravan  or  convoy),  more  complicated  missions,  such  as  reconnaissance,  require  a  great 
deal  more  attention  from  controllers  in  turn  creating  higher  levels  of  workload.  Further  complications  occur 
when  multiple  operators,  each  controlling  one  or  several  robots,  must  collaborate  and  coordinate.  Researchers 
at  UCF  investigated  these  issues  using  a  Scale  Military  Operations  in  Urban  Terrain  (MOUT)  facility 
constructed  on  site  at  UCF. 

3.2  Approach 

A  Scale  (1:35)  MOUT  facility,  representing  a  prototypical  Middle  Eastern  urban  and  suburban  environment, 
was  constructed  with  the  express  purpose  of  conducting  teaming  and  span-of-control  research  for  UGVs.  The 
MOUT  facility  has  the  capability  to  simulate  the  simultaneous  use  of  two  separate  high  mobility  multi¬ 
wheeled  vehicle-sized  UGVs.  For  reconnaissance  missions,  the  UGVs  each  have  two  independently 
controlled,  radio-transmitter  cameras  providing  images  to  remotely  located  controllers.  The  first  of  these 
cameras  provides  a  fixed  forward  ‘driver’  view  that  always  faces  in  the  direction  the  vehicle  faces  and  cannot 
be  moved  independent  of  the  UGV.  The  second  camera  was  in  a  top  turret  mounted  unit  which  can  be  rotated 
independently  of  UGV  movement  to  serve  as  a  reconnaissance  unit. 

Using  the  facility,  participants  completed  reconnaissance  missions  in  teams  of  either  one  or  two  controllers 
using  either  one  or  two  UGVs.  In  addition  to  these  variables,  a  component  of  vehicle  reliability  (high  vs.  low) 
was  added  as  a  between  subject  variable  and  task  difficulty  (high  vs.  low)  as  a  within  subject  variable  creating 
a  mixed  model  2  x  2  x  (2  x  2)  design.  Participants  were  initially  shown  a  short  training  session  that  covered 
both  UGV  control  and  target  identification  (via  BLUFOR  and  OPFOR  handbooks)  and  then  were  to  complete 
four  20-minute  performance  measure  sessions.  In  between  each  session,  workload  measures  were  taken  and 
various  spatial  tests  were  administered.  Participants  were  measured  on  the  number  of  correct  identifications 
made  of  potential  targets  made  during  each  of  the  four  trials.  Figure  4  provides  an  example  of  what  controllers 
may  have  seen  via  the  vehicle  mounted  cameras. 

3.3  What  Was  Learned 

Mentioned  above  was  the  hope  that  one  soldier  would  be  able  to  effectively  control  a  number  of  UGVs  in 
various  military  situations.  Unfortunately,  this  does  not  appear  to  be  the  case.  Teams  consisting  of  two 
participants  far  out  performed  (by  an  average  of  70%)  teams  consisting  of  one  participant  in  terms  of  number 
of  target  items  correctly  identified.  In  the  worst  conditions,  with  unreliable  vehicles  performing  in  the  most 
difficult  task  scenarios  teams  with  two  members  performed  190%  better  than  their  individual  counterparts. 
Indeed,  the  teams  that  consisted  of  one  controller  handling  two  UGVs  showed  the  poorest  performance  of  all 
the  possible  team  configurations. 
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Figure  4:  The  Forward  View  UGV  Camera  Showing  insurgents  in  the  Crowd  with  Weapons  Aimed. 


The  addition  of  a  second  UGV  had  at  best  no  impact  on  target  identification  performance  and  at  worst  actually 
produced  a  reduction  in  performance.  For  teams  of  two  controllers,  this  lack  of  increased  performance  was 
likely  due  to  the  high  use  of  an  “overwatch”  technique  employed  by  the  controllers.  Using  this  technique, 
UGVs  would  simply  leapfrog  one  another  from  waypoint  to  waypoint  without  utilizing  the  UGVs  in  different 
routes.  An  example  of  one  of  these  routes  can  be  seen  in  Figure  5.  For  teams  with  only  one  controller  the 
addition  of  a  second  vehicle  only  reduced  overall  performance,  sometimes  by  as  much  as  50%. 


Figure  5:  Example  of  ‘Overwatch’  Route  Plan  for  Multiple  UGVs.  Notice  both  vehicle 
paths  (the  brown,  purple,  orange  and  green  lines)  follow  exactly  the  same  route. 
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Admittedly,  participants  in  this  study  were  naive  to  the  identification  task,  which  surely  contributed  to  the  low 
overall  identification  rate  (roughly  20%  of  possible  targets).  Studies  are  currently  planned  that  will  not  only 
increase  the  levels  of  training  for  participants  but  also  will  utilize  participants  more  familiar  with  the  subject 
matter. 

3.4  Advantages  and  Disadvantages 

Use  of  the  Scale  MOUT  facility  carried  with  it  several  advantages.  One  of  the  largest  of  these  advantages  is 
the  high  fidelity  environment  provided  to  participants.  The  sense  of  realism  created  by  having  an  actual 
physical  environment,  and  not  a  computer  simulated  one,  ensure  that  participants  had  a  feeling  of  consequence 
to  their  actions.  For  example  if  a  participant  ran  a  vehicle  into  a  building  or  person,  real  damage  would  take 
place  and  the  UGV  would  not  simply  drive  through  the  structure.  This  high  level  of  fidelity  meant  that 
participants  would  control  UGVs  with  more  responsibility  than  they  might  have  in  a  virtual  environment, 
taking  care  to  drive  more  slowly  and  be  more  conscious  of  their  surroundings. 

In  addition,  use  of  the  Scale  MOUT  facility  gave  researchers  flexibility  in  the  environment  which  could  be 
reconfigured  quickly  due  to  its  component  design.  The  entire  facility  is  created  by  arranging  2’  x  2’  pallets 
with  various  building  vehicles  and  figures  on  them  that  can  quickly  be  rearranged  to  alter  the  environment. 

The  facility  is  not  without  its  disadvantages,  however.  The  use  of  a  large  number  of  radio  transmission  devices 
(vehicle  remote  controls,  camera  transmitters,  etc.)  means  that  interference  can  become  an  issue.  UGV 
controls  can  lose  signal  and  vehicle  control  can  be  lost  as  well  as  camera  signal  resulting  in  a  scrambled  image 
being  transferred.  While  this  is  a  real  world  problem  that  soldiers  would  likely  have  to  deal  with  in  actual 
military  operations,  it  is  not  an  ideal  research  condition,  causing  disruptions  during  experimental  trials. 

Finally,  it  should  be  noted  that  many  people  were  required  to  run  the  Scale  MOUT  facility  efficiently.  For  an 
example  session,  up  to  four  experimental  personnel  are  needed.  The  experimenter  interacts  with  the 
participants,  while  up  to  two  confederates  simulate  the  automation  of  the  UGVs.  Finally,  one  research 
assistant  is  required  as  technical  support  help  for  any  other  issues  (a  role  which  in  most  cases,  however,  could 
also  be  fulfilled  by  one  of  the  confederates). 


4  A  PURPOSE-BUILT  SYNTHETIC  TASK  ENVIRONMENT 

4.1  Background  and  Purpose 

Robotic  technology  will  be  a  vital  component  of  future  combat.  The  future  Soldier  will  conduct  Traditional 
military  tasks’  such  as  scanning  for  threats,  engaging  targets,  and  conducting  communications;  additionally, 
the  Soldier  will  have  to  control  or  monitor  the  unmanned  system(s)  and  process  information  returned  from  the 
unmanned  system(s).  Robotic  control  will  occur  via  an  interface.  Parasuraman  and  Cosenzo  (Cosenzo, 
Parasuraman,  Barnes,  &  Novak,  2006)  developed  simulations  to  evaluate  human  interactions  with  robotic 
systems  particularly  how  Soldiers’  multi-tasking  requirements  affect  their  ability  to  successfully  complete 
their  missions.  More  specifically,  their  goal  is  to  investigate  if  and  how  adaptive  automation  should  be 
implemented  to  support  human  control  of  multiple  robotic  systems  from  a  single  control  unit. 

4.2  Approach 

ARL,  HRED,  in  collaboration  with  George  Mason  University  created  a  simulation  testbed  which  emulates  the 
essential  robotic  tasks  (see  Cosenzo  et  al.,  2006  for  details).  The  program.  Robotic  NCO,  was  based  on  an 
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existing  prototype  operator  control  units  (OCU)  designed  by  MA&D  and  TARDEC.  Robotic  NCO  allows 
control  of  both  the  task  types  and  levels  of  automation  available  to  the  operator.  Robotic  NCO  is  a  multi-task 
environment  that  requires  participants  to  complete  three  tasks  simultaneously:  (a)  Respond  to  targets 
encountered  by  a  UAV  by  using  the  mouse,  (b)  respond  to  potential  obstacles  and  waypoints  encountered  by  a 
UGV  via  the  mouse,  and  (c)  respond  to  audio  communications  via  the  keyboard  (Figure  6).  The  program  was 
run  using  a  Dell  PC,  keyboard,  mouse,  and  monitor. 


Figure  6:  Screen  Shot  of  the  Robotic  NCO  Simuiation. 


4.3  What  was  Learned 

The  purpose  of  the  first  experiment  with  the  Robotic  NCO  software  program  was  to  evaluate  the  effects  of 
task  difficulty  on  performance,  workload,  and  SA.  In  the  second  experiment,  enemy  movement  occurred  at 
unpredictable  times  causing  an  icon  on  the  SA  map  to  change  its  location.  The  operator  was  required  to  report 
the  change  in  enemy  position.  Preliminary  results  indicate  that  the  change  detection  task  was  extremely 
sensitive  to  manipulation  of  multi-tasking  difficulty.  The  simulation  research  has  identified  several  possible 
automation  tasks  including  SA  aids,  target  identification,  and  robotic  control.  In  future  work,  the  researchers 
will  be  simulating  automation  strategies  in  the  Robotic  NCO  and  investigating  their  effects  on  alleviating 
workload  and  improving  primary  task  performance.  The  series  of  experiments  will  help  the  researchers 
understand  the  generic  tasking  environment  for  mounted  soldiers  and  will  further  their  understanding  of 
adaptive  (dynamically  changing)  and  static  automation  strategies  for  workload  reduction  during  mounted 
scenarios.  The  advantage  to  Robotic  NCO  is  that  it  will  allow  the  initial  studies  to  be  tightly  controlled 
resulting  in  efficient  utilization  of  limited  resources.  As  more  is  learned  about  Soldier  performance,  other 
simulation  capabilities  will  be  investigated.  For  example,  the  researchers  intend  to  use  a  simulated  TCU  from 
an  ongoing  Army  robotic  program  and  eventually  conduct  field  studies  to  validate  and  extend  their  initial 
findings  with  actual  robotic  systems. 
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5  COMBINING  OFF-THE-SHELF  WITH  PURPOSE-BUILT  SOFTWARE 

5.1  Background  and  Purpose 

The  goal  of  this  research  was  to  examine  if  gunners  in  a  U.S.  Army  PCS  vehicle  such  as  the  MCS  were  able  to 
effectively  maintain  local  security  (i.e.,  perform  their  gunner’s  tasks)  while  managing  their  unmanned  assets 
and  how  individual  difference  factors  such  as  attentional  control  and  spatial  ability  impacted  their 
performance  (Chen  &  Joyner,  2006;  Mitchell  &  Chen,  2006).  Mitchell  (2005)  examined  workload  for 
possible  crew  configurations  of  MCS  crew  members  using  Improved  Performance  Research  Integration  Tool 
(IMPRINT)  modeling,  the  gunner  is  the  most  viable  option  for  controlling  robotic  assets  compared  to  the 
other  two  positions  (i.e.,  commander  and  driver).  She  found  that  the  gunner  had  the  fewest  instances  of 
overload  and  could  assume  control  of  the  robotic  tasks.  However,  she  also  discovered  there  were  instances  in 
the  model  when  the  gunner  dropped  his  or  her  primary  tasks  of  detecting  and  engaging  targets  to  perform 
robotic  control  tasks,  which  could  be  catastrophic  for  the  team  and  mission  during  a  real  operation. 

This  current  research  tried  to  verify  the  modeling  project’s  analytical  results  and  examined  if  the  gunner  could 
effectively  detect  targets  in  his  or  her  immediate  environment  while  operating  robotic  assets  in  a  remote 
environment.  Past  research  in  dual  task  environments  suggests  performance  decrements  when  both  tasks 
involved  focal  vision  as  in  detecting  road  hazards  while  operating  in-vehicle-devices  (Horrey  &  Wickens, 
2004),  when  the  number  of  the  monitored  displays  increased  (Murray,  1994),  and  as  the  size  of  the  search  set 
increased  (Scanlan,  1977).  In  the  current  study,  it  was  expected  that  performance  would  be  worse  in  the 
concurrent  task  conditions  because  of  the  divided  visual  attention,  and  that  the  gunner’s  task  performance 
would  further  degrade  when  the  robotic  tasks  became  more  challenging  (i.e.,  when  more  than  mere 
monitoring  was  needed). 

5.2  Approach 

Chen  and  Joyner  (2006)  simulated  a  mounted  environment  that  captured  some  of  the  generic  capacities  of 
future  mounted  systems  and  conducted  an  experiment  to  examine  the  workload  and  performance  of  the 
combined  position  of  gunner  and  robotic  operator.  The  experimental  conditions  included  a  gunner  baseline 
(Gunner  Baseline  condition)  and  concurrent  task  conditions  where  participants  simultaneously  performed 
gunnery  tasks  and  one  of  the  following  tasks:  monitor  a  UGV  via  the  video  feed  (Monitor  condition),  manage 
a  semi-autonomous  UGV  (UGV  condition),  and  teleoperate  a  UGV  (Teleop  condition).  Participants  also 
performed  a  tertiary  communication  task  concurrently.  The  Simulation  Integration  Laboratory  ([SIL] 
specifically  the  TCU  developed  by  ARLs  Robotics  Collaborative  Technology  Alliance  (RCTA),  was  used  for 
the  robotic  tasks  (this  component  has  also  been  used  at  George  Mason  University  in  some  of  their  studies). 
The  gunnery  component  was  implemented  using  an  additional  screen  and  controls  to  simulate  the  out-of-the- 
window  view  and  firing  capabilities  (Figure  7). 
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Figure  7:  TCU  (left)  and  Gunnery  Station  (Gunner’s  Out-the-Window  View)  (right). 


The  main  interface  of  the  SIL  is  the  TCU  which  is  where  the  operator  tasks  and  monitors  the  robotic  asset. 
With  the  SIL  various  robotic  assets  can  be  simulated,  large  UGVs,  UGSs,  and  UAVs.  The  simulation  software 
is  used  to  populate  the  reconnaissance  area  with  multiple  entities,  friendly,  enemy,  or  neutral.  In  a  typical 
scenario,  operators  are  asked  to  move  assets  from  a  start  point  through  all  checkpoints  to  the  release  point  and 
to  identify  targets  along  the  way.  Subjects  are  first  provided  a  map  with  waypoints  and  required  to  set  up 
reconnaissance  scans,  which  include  the  terrain  inside  areas  of  interests.  In  this  environment,  we  plan  to 
investigate  the  effects  of  number  of  robotic  assets  and  reliability  on  performance  in  the  SIL.  The  research  will 
transition  results  obtained  with  the  Robotic  NCO  simulation  and  validate  those  findings  in  the  SIL,  a  higher 
fidelity  environment. 

5.3  What  Was  Learned 

Results  showed  that  gunner’s  target  detection  performance  degraded  significantly  when  he  or  she  had  to 
concurrently  monitor,  manage,  or  teleoperate  a  UGV  compared  to  the  baseline  condition  (gunnery  task  only). 
The  gunner’s  performance  was  significantly  better  in  the  Monitor  condition  than  the  UGV  condition,  which  in 
turn  was  significantly  better  than  in  the  Teleop  condition.  For  the  robotic  tasks,  there  were  significant 
differences  among  the  Monitor,  UGV,  and  Teleop  conditions  in  target  detection  performance,  with  the  UGV 
being  the  lowest  (only  53%  were  detected).  Participants  with  higher  perceived  attentional  control  (PAG) 
(measured  by  a  self-assessment  survey)  performed  better  on  the  tertiary  communication  task  in  the  more 
challenging  robotic  task  conditions,  although  they  performed  at  a  similar  level  on  their  gunnery  and  robotic 
control  tasks  as  those  with  lower  a  PAG.  Participants’  perceived  workload  increased  almost  linearly  in  order 
from  the  Gunner  Baseline,  Monitor,  UGV,  and  to  the  Teleop  condition.  Figure  8  shows  participants’  gunnery 
task  performance  and  perceived  workload. 
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Figure  8:  Gunnery  Task  Performance  and  Perceived  Workioad. 

5.4  Advantages  and  Disadvantages 

One  advantage  of  this  study  was  that  an  actual  Army  system  (the  TCU)  was  used  and  incorporated  in  a 
simulated  MCS  environment  that  was  created  by  the  researchers.  By  doing  so,  our  results  can  be  readily 
transferable  to  the  PCS  community.  In  fact,  the  designers  of  the  MCS  are  currently  evaluating  the  addition  of 
the  ARV  to  the  MCS  crew  and  have  incorporated  the  ARL  modeling  and  experimental  data  into  their  manning 
assessment  for  system  functional  review  (Mitchell  &  Chen,  2006).  In  addition,  the  Army  Unit  of  Action  Main 
Battlelab  has  recognized  that  allocation  of  unmanned  assets  needs  to  be  evaluated  across  all  the  crewmembers 
in  the  MCS  Platoon  (Mitchell  &  Chen,  2006).  However,  there  were  also  some  disadvantages  associated  with 
using  an  actual  Army  system  that  is  still  under  development.  The  researchers  were  not  able  to  obtain  the 
source  code  for  the  TCU  and,  therefore,  the  manipulation  of  the  robotic  tasks  and  data  collection  efforts  were 
limited. 


6  GENERAL  CONCLUSIONS 

Modeling  and  simulation  provide  a  cost  effective  means  for  training  Soldiers,  developing  doctrine  and  tactics, 
and  evaluating  systems  (National  Research  Council,  1997).  More  specifically,  they  can  be  used  to  train  and 
test  participants  for  a  particular  mission  when  access  to  the  actual  robotic  systems  is  not  feasible.  Simulations 
can  also  be  used  as  prototypes  for  future  military  systems.  Immersive  simulations  can  be  developed  that  allow 
the  participant  to  enter  and  navigate  a  particular  environment  increasing  the  realism  of  the  participant’s 
experience.  A  broad  range  of  entities  (e.g.  tanks,  aircraft,  and  people)  can  be  represented  in  a  simulation  and 
their  behaviors  can  be  programmed  and  modified  depending  on  the  scenario. 

The  above  examples  indicate  that  the  range  of  virtual  environments  in  not  constrained  by  their  cost  but  rather 
by  the  creativity  of  the  experimenters  who  create  them.  Even  the  most  expensive  of  the  above  examples,  was 
cost  effective  when  compared  to  even  limited  field  exercises  (Barnes,  Hunn  &  Pomranky,  2006).  The 
versatility  of  the  reviewed  research  paradigms  also  has  the  advantage  of  generalizing  too  many  real  world 
situations  enhancing  our  ability  to  develop  performance  models  that  capture  the  essence  of  HRI  behaviors. 
These  models  can  be  used  to  begin  the  process  of  developing  Soldier-centered  designs,  tactics,  and  procedures 
for  PCS  aerial  and  ground  robotic  systems.  Such  models  will  never  be  perfect  and  they  must  be  validated  but 
they  will  be  invaluable  initial  estimates. 


RTO-MP-HFM-136 


10-13 


Understanding  Soldier  Robot  Teams  in  Virtual  Environments 


ORGANIZATION 


In  the  future,  we  will  develop  increasingly  realistic  simulations  as  we  understand  the  efficacy  of  adaptable  or 
adaptive  options  in  multi-tasking  environments  involving  supervision  and  control  of  robotic  assets.  As  the 
fidelity  of  the  simulations  improves,  the  ability  to  transition  the  human  robotic  technologies  into  fieldable 
systems  becomes  more  likely  as  well. 


7  REFERENCES 

Allender,  L.,  McDermott,  P.  L.,  Luck,  J.,  &  Fisher,  A.  (2006).  Team  communication  with  and  without  aids  for 
transmitting  remote  information.  Manuscript  submitted  for  publication. 

Barnes,  M.  J.,  Hunn,  B.  P.,  &  Pomranky,  R.  A.  (2006).  Modeling  and  operator  simulations  for  early 
development  of  Army  unmanned  vehicles:  methods  and  results.  In  N.  J.  Cooke,  H.  L.  Pringle,  H.  K. 
Pedersen,  &  O.  Connor  (Eds.),  Human  factors  of  remotely  operated  vehicles  (pp.  59-71).  Amsterdam: 
Elsevier. 

Chen,  J.  Y.  C.,  &  Joyner,  C.  T.  (2006).  Concurrent  performance  of  gunner  and  robotic  operator’s  task  in  a 
multitasking  environment.  Manuscript  submitted  for  publication. 

Cosenzo,  K.,  Parasuraman,  R.,  Barnes,  M.,  &  Novak,  A  (2006).  Adaptive  automation  for  robotic  military 
systems.  Manuscript  submitted  for  publication. 

Horrey,  W.,  &  Wickens,  C.  (2004).  Focal  and  ambient  visual  contributions  and  driver  visual  scanning  in  lane 
keeping  and  hazard  detection.  Proceedings  of  the  Human  Factors  and  Ergonomics  Society  48^^  Annual 
Meeting  (pp.  2325-2329). 

McDermott,  P.  L.,  Luck,  J.,  Allender,  L.,  &  Fisher,  A.  (2005).  Effective  human  to  human  communication  of 
information  provided  by  an  unmanned  vehicle.  Proceedings  of  the  Human  Factors  and  Ergonomics 
Society  49^^  Annual  Meeting. 

Luck,  J.,  McDermott,  P.  L.,  Allender,  L.,  &  Fisher,  A.  (2006).  Advantages  of  co-location  for  effective  human 
to  human  communication  of  information  provided  by  an  unmanned  vehicle.  Manuscript  submitted  for 
publication. 

Mitchell,  D.  (2005).  Soldier  workload  analysis  of  the  mounted  combat  system  (MCS)  platoon’s  use  of 
unmanned  assets  (ARL-TR-3476).  Aberdeen  Proving  Ground,  MD:  U.S.  Army  Research  Laboratory. 

Mitchell,  D.,  &  Chen,  J.  Y.  C.  (2006).  Impacting  system  design  with  human  performance  modeling  and 
experiment:  Another  success  story.  Manuscript  submitted  for  publication. 

Murray,  S.  (1994).  Human  performance  studies  for  control  of  multiple  remote  systems  (Tech.  Rep.  NRaD 
1658).  San  Diego,  CA:  SPAWAR. 

National  Research  Council,  Committee  on  Modeling  and  Simulation.  (1997).  Modeling  and  simulation: 
Linking  entertainment  and  defense.  National  Academies  Press,  Washington,  D.C. 

Scanlan,  L.  A.  (1977).  Target  acquisition  in  realistic  terrain.  Proceedings  of  Human  Factors  and  Ergonomics 
Society  2 Annual  Meeting  (pp.  249-253). 


10-14 


RTO-MP-HFM-136 


Technology  for  Human-Robot  Interaction  and  Teaming: 
Understanding  Soldier  Robot  Teams 
in  Virtual  Environments 


Human  Research  and  Engineering  Directorate 


Human  Factors  Medicine  Panel  (HFM-136) 
NATO  workshop 

Virtual  media  for  Military  Applications 
United  States  Military  Academy 
June  13-15 
West  Point,  NY 


Michael  J.  Barnes 

U.S.  Army  Research  Laboratory 
2520  Healy  Ave,  Ste  1172 
Fort  Huachuca,  AZ  85613-7069 
Keryl  A.  Cosenzo 
U.S.  Army  Research  Laboratory 
Building  459 

Aberdeen  Proving  Ground,  MD  21005-5425 
Florian  Jentsch 
University  of  Central  Florida 
Department  of  Psychology 
12424  Research  Parkway,  Ste  408 
Orlando,  FL  32826 
Jessie  Y.  C.  Chen 
U.S.  Army  Research  Laboratory 
12423  Research  Parkway 
Orlando,  FL  32826 
Patricia  McDermott 
Microanalysis  &  Designs 
4949  Pearl  E.  Circle,  Ste  300 

Boulder,  CO  80301  , 


Background 


Human  Research  and  Engineering  Directorate 


•  Purpose  :  To  understand  Soldier  robot 
interactions  and  teaming  performance  in  order 
to  support  the  Army  Technologhy  Objective 
(ATO):  Collaborative  Robotics 


•  Future  robotic  systems  and  Soldier 
environments  and  tactics  are  evolving  and 
cannot  always  be  duplicated  efficientTy  in  the 
real  world. 


•  Virtual  media,  games,  scale  models,  and  other 
software  environments  will  give  us  the  flexibility 
to  ask  important  questions  concerning  systems 
that  do  not  yet  exist 

•  The  diversity  of  these  approaches  will  allow  the 
researchers  to  understand  Soldier  interactions 
with  aerial  and  ground  robotic  systems  in  a  wida 
variety  of  military  scenarios. 


Overview  and  Comparison 


Human  Research  and  Engineering  Directorate 


Project 

Description 

Investigations 

University  of 

Central  Florida  (UCF) 

1/35  Iraqi  City 
Scaled  robots  - 
recon  mission 

Teaming-  multiple 
robots  with  multiple 
Soldiers 

Microanalysis 

&  Designs  (MAAD)  w. 
ARL 

Ravenshield 
gaming  tool 
Rescue  mission 

Aerial  and  ground 
robots  to  assist  in 
recovering  downed 
crew 

George  Mason 
University  &  ARL 

Robotics  NCO 

workstation 

Effects  of  adaptive 
automation  for  robot 
related  multitasking 
scenarios 

ARL  w.  UCF 

Gunner 
workstation  for 
future  mounted 
systems 

Measure  workload 
and  targeting  for 
adding  robotic  tasks 
in  future  system  ^ 

Teaming  in  a  Realistic  Urban  Setting 
1/35th  Scale  -  Jentsch  &  al.  UCF 


Human  Research  and  Engineering  Directorate 


Statement  of  the  Problem  Approach 


Need  to  understand  and  enhance 
the  interactions  of  multiple  soldiers 
with  multiple,  semi-autonomous 
unmanned  ground  vehicles 
(UGVs),  especially  during  military 
operations  in  urban  terrain 


Sample  Study: 

-  Compare  teams  of  two  operators 
to  single  operators,  under  different 
conditions  of  workload,  operator 
intervention  required,  and  number 
of  operators  available 

Method: 

-  Developed  simulation  environment 

-  Built  and  instrumented  the 
simulated  UGVs 

-  Developed  scenarios 

-  Trained  confederates  for  “man- 
behind-the-curtain”  operation  of 
the  simulated  UGVs 

-  Collected  data  from  participants 
(Students  and  ROTC  Cadets) 
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Description  of  the  UCF 
Simulation  Environment 


Human  Research  and  Engineering  Directorate 


•  The  UCF  TPL  scale 
MOUT  facility  consists  of  a 
1/35  scale  representation  of 
a  Middle  Eastern 
environment  including  four 
distinct  terrain  areas 
(Desert,  Residential,  Old 
City,  Modern  City) 

•  Representing  and  area  of 
approximately  4-6  city 
blocks,  routes  of  several 
miles  can  be  achieved  with 
the  use  of  meandering 
streets 

•  Capable  of  employing  2 
UGVs  simultaneously  each 
with  independent  control 
and  visual  feedback 


5 


UCF  Study  Results  and  Assessment 


Human  Research  and  Engineering  Directorate 


Having  a  second  operator  available  always 
improved  performance: 

-  On  average,  teams  did  70%  better  than  individuals 

-  When  the  conditions  were  difficult,  however,  two 
operators  performed  up  to  190%  better  than  individuals 

The  addition  of  a  second  UGV  to  this  RSTA  task 
was  at  best  marginally  useful: 

-  Teams  of  two  showed  slightly  better  performance  with 
two  UGVs  in  easy  scenarios 

-  When  single  operators  had  to  control/supervise  two 
UGVs,  performance  typically  suffered,  at  worst  by  up 
to  50% 

-  Both  teams  and  individuals  typically  employed  two 
vehicles  “in  trail”  or  in  an  “overwatch”  manner 


Study  clearly  demonstrated  the  feasibility  and  utility 
of  studying  HRI,  especially  teaming  of  multiple 
soldiers  with  multiple  robots,  through  the  use  of  a 
physical  scale  environment: 

-  Future  extensions:  2""^  scaled  environment  (“Bosnia”) 
for  training,  linkage  to  hi-fidelity  operator/control  station 

-  Since  the  original  study,  three  more  studies  have  been 
completed  using  the  scale  MOUT  environment 


SI  One 
Vehicle 
■  Two 
Vehicles 


One  Operator  Two  Operators 


Ravenshield:  MAAD  and  ARL 
Black  Hawk  Down  rescue  mission 


•  Researchers-  Luck,  McDermott  (MAAD)  &  Allender  (ARL) 

•  How  can  teams  effectively  use  the  information 

provided  by  unmanned  vehicles  (UVs)? 

-  Is  performance  different  with  an  aerial  vehicle  than  a 
ground  vehicle? 

-  How  useful  are  collaborative  technologies  such  as  an 
electronic  map  or  the  ability  to  transmit  images? 

-  What  are  the  implications  of  having  the  information 
manager  at  a  rearward  command  center? 

-  If  being  co-located  is  superior,  why? 

•  Used  Tom  Clancy’s  Ravenshield  (by  UbiSoft™) 

-  Two  experiments 
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-  US  Marines  participated  in  two-person  teams 


Ravenshield 


Human  Research  and  Engineering  Directorate 


•  Commercially  available  video  game  in 
which  participants  could  complete 
“Blackhawk  Down”  missions 


•  Network  capabilities  allowed  multiple 
players  in  the  same  environment 

•  In  Experiment  1,  the  “ghost”  mode 
simulated  UAVs  (unmanned  aerial 
vehicle)  and  UGVs  (ground) 


View  from  UGV 


*  In  Experiment  2,  the 
appearance  of  the  UV  was 
modified  and  was  another 
“player”  in  the  game 


UV  from  Player’s  perspective 


Map  with  player  locations 


Results:  Luck,  McDermott  &  Allender 

MAAD  &  ARL 
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-  Teams  were  taster  and  detected  less  otten  by  the  enemy 
with  the  UGV  than  the  UAV 


-  Participants  preferred  the  flexibility  of  the  UAV 

-  There  were  advantages  being  co-located 

-  Being  able  to  transmit  images  seemed  to  compensate  for 
the  fact  that  teams  could  not  talk  face  to  face 

•  Advantages 

-  Simulated  UV  was  relatively  easy  to  operate  -  allowed  us  to 
focus  on  communication  versus  control  issues 

-  Environment  was  authorable;  Maps  and  help  available  on 
Internet;  inexpensive 

•  Disadvantages 

-  Workarounds  required  to  get  the  game  to  work  as  intended 

-  Not  designed  to  collect  the  type  of  data  needed  ^ 


Robotics  NCO  Simulations  :  Adaptive 
Automation  -  Cosenzo  &  Parasuraman 

m  ji^GEORqE 

iMakini 


Human  Research  and  Engineering  Directorate 


■  university 


Purpose  and  Methods 

Measure  the  cognitive  task  difficuity  in  a  muititasking  environment 
representing  the  mounted  operator  monitoring  and  controiling  aeriai 
and  ground  robotic  missions 

Develop  adaptive  automation  techniques  to  reduce  performance 
and  workload  problems 


Exp  1  and  2  -  measure  multitasking  performance  -  change  blindness, 
response  time 

Exp  3-  in  progress  -effects  of  adaptive  automation 
Advantages  and  Disadvantages 

'Experimental  setting  that  captures  the  essential  military  and  workload 
component  of  the  robotic  environment  allowing  manipulation  of  key 
variables. 


•Simulation  cannot  capture  the  stress,  complexity  or  workload  of 
combat  environment.  io 


Simulation  environment 
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Highly  integrated  multitask  environment;  actions  that  occur  in  one 
task  affect  other  tasks  and  views  or  require  additional  actions  by  the 

operator. 


Robotic  NCO  Simulation 


Zoomecl  In  UAVView 
Possible  Target 


i 

i 


Map  with  i 

UAV  and  UQV  Path  % 
Some  Friendly  Units  [ 

E 


c> 


UAV  Status 


Menu 


c 


Communications  ^  ^ 
Screen  ! 


Sample  results:  Response  Time 
ommunications-  Cosenzo  &  Parasuraman 

|m  jfGEORGEl 

Ma^n 
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UNIVERSITY 


High-priority  but  infrequently  occurring  communications  pose  a 
particularly  high  monitoring  load  on  the  operator,  as  suggested  by 
recent  studies  of  vigilance  and  monitoring  in  semi-automated  systems. 


Low  Comms 


1900 


Low  UGV  High  UGV 


■  Low  UAV 

■  High  UAV 


High  Comms 


UAV  Targets  x  UGV  Requests  x  COMMS,  Wilk’s  A  (2,15)  =  3.38,  p  =  .06. 


Purpose  &  Method:  Chen  &  Joyner 
Simulation  of  Future  Systems 


i 


m 


•  Purpose:  Understand  multitasking  environment  for  the 
gunner  in  future  mounted  systems  -  communications, 
UGV  targeting  and  baseline  gunnery  tasks-  Verify 
workload  estimates  of  25  soldiers  from  Ft.  Knox 

•  Participants:  20  students  from  Univ.  of  Central  Florida 

•  Training  session  (about  2  hours) 

-  Preliminary:  Informed  consent,  demographics  survey,  attentional 
control  survey,  spatial  ability  tests  (Cube  Comparison  Test  and 
Spatial  Orientation  Test) 

-  Training:  Self-paced  tutorial  and  exercises 

•  Experimental  session  (about  3  hours) 

•  Post-experiment  surveys 

-  NASA-TLX  (workload)  after  each  condition 

-  SSQ  (Simulator  Sickness  Questionnaire)  ^ 

-  Usability  questionnaire 


Apparatus-  Robotics  tactical  control  unit  (TCU)  & 
Gunnery  Station  (Chen  &  Joyner) 


Dual  tasks:  a.)  Baseline-primary:  Local  security  (360deg)  for  the 
gunnery  station  b.)  Remote  targeting  with  the  TCU 
Experimental  conditions:  (1)  Baseiine  (No  TCU),  (2)  Monitor  TCU  +1, 
(3)  Waypoint  controi  of  the  UGV  +  1  and  (4)  Teleops  of  the  UGV  +1 


Targets  Detected 


Results  of  primary  task 
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Target  Detection-  Overall 


Perceived  Workload 


14.00- 


12.00- 


10.00- 


8.00- 


Gunner  BL 


Gunner-Baseline  Monitor  UGV  Teleop 


Condition 


Condition 


Advantages  &  Disadvantages 
Chen  &  Joyner-2006 
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•  Advantages: 

-  an  actual  Army  system  (the  TCU) 

-  results  can  be  readily  transferable  to  the  FCS  community 

•  designers  of  the  future  systems  are  currently  evaluating  the  addition  of 
the  ARV  to  the  crew  and  have  incorporated  the  ARL  modeling  and 
experimental  data  into  their  manning  assessment  for  system  functional 
review  (Mitchell  &  Chen,  2006) 

•  the  Army  Unit  of  Action  Main  Battlelab  has  recognized  that  allocation  of 
unmanned  assets  needs  to  be  evaluated  across  all  the  crewmembers  in 
the  Mounted  Combat  System  Platoon  (Mitchell  &  Chen,  2006). 

•  Disadvantages 

-  TCU  is  still  under  development. 

•  The  researchers  were  not  able  to  obtain  the  source  code  for  the  TCU  and, 
therefore,  the  manipulation  of  the  robotic  tasks  and  data  collection  efforts 
were  limited. 


Conclusions 
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•  Virtual  representations  allow  creative  ways  to 
capture  future  environments 

•  1/35th  scale  city  was  a  flexible  means  to  capture 
the  dynamics  of  near  term  future  robotic  mission 

•  Ravenshield  game  was  not  as  flexible  but 
captured  the  essence  of  a  future  rescue  mission 
with  robotic  assistance 


•  Robotics  NCO  permitted  experimentally  controlled 
manipulations  of  a  UAV/UGv  multitasking 
environment  to  evaluate  adaptive  automation 

•  The  gunner  station  and  TCU  permitted  simualtion 
of  of  a  future  mounted  mission  with  realistic 
displays  and  controls  necessary  to  estimate  future 
peitormance  and  workload  issues  resulting  froi;p 
robotic  taskings 


